-
Notifications
You must be signed in to change notification settings - Fork 201
Android SDK build scripts #467
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
* Build Android image * Checkout without ssh * Retry build if it fails * Swift 6.1 Release Dockerfiles (swiftlang#456) * Change binutils-gold package dependency on Debian 12 to binutils (swiftlang#457) * Update installed packages after nightly platform expansion (swiftlang#458) * update nightly-6.1 dependencies * update nightly-main dependencies * fix ubuntu images * Android SDK build * Android SDK build * Android SDK build * Android SDK build * Android SDK build * Android SDK build * Fedora 41 Dockerfile (swiftlang#464) * Build Android image * Build Android image * Swift 6.1 Release Dockerfiles (swiftlang#456) * Change binutils-gold package dependency on Debian 12 to binutils (swiftlang#457) * Build Android image * Build Android image * Build Android image * Android SDK build * Android SDK build * Android SDK build * Android SDK build * Android SDK build * Android SDK build * Android SDK build * Android SDK build * Android SDK build * Android SDK build * Android SDK build * Android SDK build * Android SDK build * Android SDK build * Android SDK build * Android SDK build * Android SDK build * Android SDK build * Android SDK build * Android SDK build * Android SDK build * Android SDK build * Android SDK build * Android SDK build * Android SDK build * Android SDK build * Android SDK build * Android SDK build * Android SDK build * Android SDK build * Android SDK build * Android SDK build * Android SDK build * Android SDK build * Android SDK build * Android SDK build * Swift Androd build * Swift Androd build * Swift Androd build * Swift Androd build * Swift Androd build * Swift Androd build * Swift Androd build * Swift Androd build * Swift Androd build * Swift Androd build * Swift Androd build * Swift Androd build * Swift Androd build * Swift Androd build * Swift Androd build * Swift Androd build * Swift Androd build * Swift Android build * Swift Android build * Swift Android build * Swift Android build --------- Co-authored-by: Mishal Shah <shahmishal@users.noreply.github.com> Co-authored-by: Chris McGee <87777443+cmcgee1024@users.noreply.github.com> Co-authored-by: Justice Adams <107649528+justice-adams-apple@users.noreply.github.com> Co-authored-by: Andrew Sukach <134116196+sookach@users.noreply.github.com>
* Swift Android build 6.2 * Swift Android build 6.2 * Swift Android build 6.2 * Start splitting NDK out from the rest of the SDK * Start splitting NDK out from the rest of the SDK * Swift Android build 6.2 * Swift Android build 6.2 * Swift Android build 6.2 * Swift Android build 6.2 * Swift Android build 6.2 * Swift Android build 6.2 * Swift Android build 6.2 * Swift Android build 6.2 * Swift Android build 6.2 * Swift Android build 6.2 * Swift Android build 6.2 * Swift Android build 6.2 * Swift Android build 6.2 * Swift Android build 6.2 * Swift Android build 6.2
* Swift Android build 6.2 * Swift Android build 6.2 * Add post-install script to SDK bundle * Add post-install script to SDK bundle * Add post-install script to SDK bundle * Add post-install script to SDK bundle
I've updated this PR to build the SDK without including the Android NDK. In order to accommodate this, there is now a post-install The latest result of the build for the 6.2 nightly can be seen at: https://github.com/skiptools/swift-android-toolchain/releases/tag/6.2-DEVELOPMENT-SNAPSHOT-2025-05-07-a |
To clarify what the post-install script is doing, following is a result of the output of the command with the default NDK directory:
Alternative NDK locations can be specified by overriding the
|
* Add static libraries to post-install script * Add post-install script to SDK bundle * Add post-install script to SDK bundle * Update submodules
As of 72964f5, the Android SDK is now being built in a Docker container. |
…z,max-page-size=16384
* Run the compiler validation suite for Android * Add --build-compiler option * Add --cross-compile-build-swift-tools=0 from swiftlang/swift#38441 * Build with --build-llvm=0 * Check out Yams for Swift 6.1.1 build * Add --llvm-ninja-targets-for-cross-compile-hosts=help * Install pre-requisites * Install build prerequisites * Only setup local toolchain if build-compiler is not 0 * Fix Yams version checkout * Quote arguments to build scripts * Permit empty host-toolchain argument in build.sh * Fix check for BUILD_COMPILER * Re-order Docker PATH to system clang is used before Swift toolchain clang * Use --host-test to skip attempt to test on connected device/emulator * Install clang in Dockerfile * Add --skip-test-linux flag to build * Remove more folders to free up space * Update patches * Add docker-specific CI variants that run the compiler validation tests * meaningless edit * Update pull_request.yml to stop the 6.1 release builds * Update pull_request.yml to really only build the full docker compiler with tests * Update README.md with meaningless edit to bump build * Update build.sh to skip testing XCTest for linux, as a handful of the linux tests fail for some reason * Update build.sh to disable building libTesting and for 16K memory pages, as both don't work yet * Update build.sh to skip testing Foundation for linux, which requires building SwiftPM from source first * Update build.sh to not build each arch in a separate build directory * Update Dockerfile to use clang 19 instead * Add self-hosted runner CI variant * Fix CI workflow syntax * Update CI runner config * Update CI runner config * Fix name of self-hosted CI runner * Re-order arch run sequence * Update CI for self-hosted runner * Build for self-hosted runner * Add swift-6.2-branch to self-hosted run matrix * Update build.sh to disable aarch64 temporarily, as we know it built fine * Update pull_request.yml to disable non-compiler builds, as they all work, and comment out self-hosted till we know armv7 works * Run compiler validation on self-hosted with increased timeout * Update build-docker to only build for armv7 * Update pull_request.yml to disable self-hosted builds temporarily * Update pull_request.yml to build the full compiler in github runners with the armv7 stdlib and tests * Update build-docker to build for aarch64 first * Update toolchain-vars.sh to use older trunk snapahot toolchain that didn't crash * Update toolchain-vars.sh to only download the latest release compiler if building the Swift compiler from source * Update build-docker to only build for armv7 again * Update toolchain-vars.sh to fix setting branch variable * Update build-docker to build for all three supported arches * Update pull_request.yml to try self-hosted runs again * Update build.sh to only install SDK components and remove linux stdlib * Tolerate missing linux folder when attempting to clean up unnecessary build artifacts * Fix extraCLIOptions in swift-toolset.json * gcpdw * Fix patch application * Update apply-patches.sh to remove unused changes * Update swift-android-testing-except-release.patch to add Testing fix * Update swift-android.patch to remove unneeded patches * Update build.sh to try and fix Testing and clean up the bundle more * Fix README.md --------- Co-authored-by: finagolfin <finagolfin@tuta.io>
…r tags Also update patches, particularly to disable failing tests from the compiler validation suite
Update patches and build the compiler from the latest commits, not an older tag
Working through this. One thing to note, Lines 79 to 83 in c95bc0c
|
I think it should be fine in theory to run the Dockerfile from another folder, but it does require some mount points to be set up (see https://github.com/swift-android-sdk/swift-docker/blob/main/swift-ci/sdks/android/build-docker#L63). I also notice that the static-linux Dockerfile doesn't build out-of-the-box with a generic command like:
Is there some different command I should run to test it? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Started going through this. I think my main concerns right now are around cloning repo's that we are already cloning with update-checkout and the patches that haven't been merged into the repos yet. Otherwise it's generally looking fine. I still need to look more closely at the build.sh
though.
} | ||
|
||
# Defaults | ||
if [[ -z "${SWIFT_VERSION}" ]]; then |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why are we downloading these separately? update-checkout
is already pulling down yams, curl, and libxml2. Do we need to update the versions there for Android? Is this just to keep those versions clean while applying patches to the versions downloaded by this script?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I didn't realize update-checkout
grabbed these too. I was just following the precedent of the static-linux SDK here:
I could likely re-use the existing yams and libxml2 checkouts, and probably curl too, but we are building and linking BoringSSL (like the static linux SDK does), so we will need to make sure that works with the version of curl that is being grabbed with update-checkout
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've removed the redundant source checkouts in swift-android-sdk#11. Thanks for spotting it! @al45tair, do you know why the static-linux SDK is checking out libxml2 and curl when it could be re-using the repositories already checked out with update-checkout
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think I wanted to have direct control over which versions get built into the Static SDK (which also let me pick new enough versions that they'd work with the other dependencies).
BTW, I'd be careful with BoringSSL. OpenSSL would be a safer choice, I think, simply because BoringSSL doesn't really have releases and so unless you collaborate directly with the BoringSSL maintainers (which is what NIO does), you won't get security fixes and so on. I'm actually thinking of switching the Static SDK over to OpenSSL for that reason — either that, or making it always grab the version NIO is currently using, since the NIO folks already watch for updates to BoringSSL.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(I wish the BoringSSL maintainers would make it slightly more explicit that using it was a bad idea, rather than merely saying they don't recommend it because of API/ABI stability. The thing I worry about is security bugs, not ABI stability.)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I had assumed/presumed that since the BoringSSL "version" being used was the "fips-20220613" tag, that that FIPS designation held some significance or potential for some security-related certification.
The other reason I stuck with it is that BoringSSL is what Android itself uses by default as of Android 6.0/API 23, even though they don't expose it as part of the NDK. I'm sure OpenSSL would be fine too, but I've read claims that BoringSSL is "optimized for Android".
making it always grab the version NIO is currently using, since the NIO folks already watch for updates to BoringSSL.
I think this would make the most sense. I wonder if we could harmonize on some source of truth for the version, and perhaps update-checkout
could start checking out BoringSSL with this tag.
Is FoundationNetwork going to be based on swift-nio/swift-nio-ssl in the future? If so, then the issue will solve itself eventually 😄
Perhaps @Lukasa has insights that would apply here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I had assumed/presumed that since the BoringSSL "version" being used was the "fips-20220613" tag, that that FIPS designation held some significance or potential for some security-related certification.
Yeah, that was my assumption too, but that's (sadly) wrong. @Lukasa, WDYT we should do here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I could likely re-use the existing yams and libxml2 checkouts, and probably curl too, but we are building and linking BoringSSL (like the static linux SDK does), so we will need to make sure that works with the version of curl that is being grabbed with update-checkout.
Note that the build-script
curl build also links against OpenSSL, but simply does so implicitly, so it only works when natively compiling and the system package is installed, then fails when cross-compiling, so we'll definitely need to add one of those SSL libraries to the build-script
config.
I wonder if we could harmonize on some source of truth for the version, and perhaps update-checkout could start checking out BoringSSL with this tag.
update-checkout
already checks out swift-crypto, which is based on BoringSSL for most platforms and has a CMake config, so maybe we can just reuse that for curl too?
@@ -21,3 +21,172 @@ jobs: | |||
name: docker-logs | |||
path: | | |||
*.log | |||
|
|||
android-build: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm concerned about running this on every PR against the swift-docker repository.
At the moment, testing on this repo takes a few minutes maximum, while I suspect that this will increase that it pretty substantially. It might be good to either limit running it to changes to the android subdirectory, or we should make this a dedicated job on Swift CI like the static Linux Musl SDK instead.
CC @shahmishal, @justice-adams-apple
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm concerned about running this on every PR against the swift-docker repository.
Yeah, actually, that android-build
CI part was intended to be expunged before this PR would be merged. 😊
It is just there so we can exercise updates to the pull request.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Okay, that sounds good to me. Either Justice or I can probably use it as a template for setting up the Jenkins job if we have questions later, so useful to have in history.
Following up on swiftlang/github-workflows#106 and swiftlang/swift#80788, this PR contains a new
swift-docker/swift-ci/sdks/android/
folder with scripts that will build and upload a full stand-alone Swift Android Swift SDK. It is modeled after the structure of the swift-ci/sdks/static-linux build scripts.Running
The top-level
./build
script installs a host toolchain and the Android NDK, and then invokesscripts/fetch-source.sh
which will fetch tagged sources for libxml2, curl, boringssl, and swift.It then applies some patches and invokes
scripts/build.sh
, which will build the sources for each of the specified architectures. Finally, it combines the NDK and the newly built SDKs into a single artifactbundle.Specifying Architectures
By default all the supported Android architectures (
aarch64
,x86_64
,aarmv7
) will be built, but this can be reduced in order to speed up the build. This can be useful, e.g., as part of a CI that validates a pull request, as building a single architecture takes around 30 minutes on a standard ubuntu-24.04 GitHub runner, whereas building for all the architectures takes over an hour.To build an artifactbundle for just the
x86_64
architecture, run:Installing and validating the SDK
The
.github/workflows/pull_request.yml
workflow will create and upload an installable SDK named something like:swift-6.1-RELEASE_android-0.1.artifactbundle.tar.gz
. The results of one of the workflow runs can be seen at https://github.com/swift-android-sdk/swift-docker/actions/runs/14603885089The workflow will also install the SDK locally and use swift-android-action to build and test various Swift packages in an Android emulator.